Papers with compact models
Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibration (2026.acl-short)
Copied to clipboard
| Challenge: | Existing methods for listwise reranking exhibit intrinsic position bias . existing methods are constrained by an inherent trade-off between efficiency and flexibility . |
| Approach: | They propose a training-free framework that mechanically decouples positional bias from ranking decisions. |
| Outcome: | a training-free framework decouples position bias from ranking decisions . evaluations show it outperforms training-based methods and outperformed expensive methods . |
Small Language Models in the Real World: Insights from Industrial Text Classification (2025.acl-industry)
Copied to clipboard
| Challenge: | With the emergence of ChatGPT, transformer-only models have significantly advanced text classification and related tasks. |
| Approach: | They propose to use prompt engineering and supervised fine-tuning methods for transformer-based text classification in industrial applications. |
| Outcome: | The proposed models perform well in a variety of industrial scenarios, including email classification, legal document categorization, and the classification of extremely long academic texts. |
LLMs for Now, Fine-Tuning for Later: An Ensemble Approach to Data Drift in Domain-Specific Tasks (2026.acl-srw)
Copied to clipboard
Yuxuan Lu, Bingsheng Yao, Shao Zhang, Yisi Sang, Yun Wang, Hansu Gu, Peng Zhang, Tun Lu, Toby Jia-Jun Li, Dakuo Wang
| Challenge: | Deploying machine learning models in domain-specific scenarios is challenged by data drift and the scarcity of expert annotations. |
| Approach: | They propose a system that combines an LLM, an AL-assisted compact model and an automatic switch module to assist the active learning process. |
| Outcome: | The proposed system achieves 96–98% switch accuracy and outperforms both models used alone. |
MiniALBERT: Model Distillation via Parameter-Efficient Recursive Transformers (2023.eacl-main)
Copied to clipboard
| Challenge: | Pre-trained Language Models (LMs) are an integral part of natural language processing but their usability is constrained by computational and time complexity and their increasing size. |
| Approach: | They propose a technique for converting knowledge of fully parameterised LMs into a compact recursive student. |
| Outcome: | The proposed models match the performance of bloated models with negligible performance losses. |
Prompt-Level Distillation: A Non-Parametric Alternative to Model Fine-Tuning for Efficient Reasoning (2026.acl-industry)
Copied to clipboard
| Challenge: | Advanced reasoning typically requires Chain-of-Thought prompting, which is accurate but incurs prohibitive latency and substantial test-time inference costs. |
| Approach: | They propose to extract explicit reasoning patterns from a Teacher model and organize them into a structured list of expressive instructions for the Student model’s System Prompt. |
| Outcome: | Evaluated using Gemma-3 4B, the proposed model improves Macro F1 scores on StereoSet and Contract-NLI while increasing LogiQA accuracy to 70%. |
On the Effects of Transformer Size on In- and Out-of-Domain Calibration (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Large pre-trained transformer language models are notoriously expensive to train . prior work has developed smaller, more compact models to reduce training costs . |
| Approach: | They propose to develop smaller, more compact transformer language models which can be calibrated in-domain . they show that smaller models can achieve competitive calibration compared to larger models . |
| Outcome: | The proposed models achieve competitive calibration and better calibration than larger models on a wide range of tasks. |
Learning from Child-directed Speech in Two-language Scenarios: A French-English Case-Study (2026.findings-eacl)
Copied to clipboard
| Challenge: | a systematic study of compact language models with limited computational resources is challenging for many research contexts and real-world applications. |
| Approach: | They extend BabyBERTa to English-French scenarios under strictly sizematched data conditions. |
| Outcome: | The proposed model extends to English-French scenarios under sizematched data conditions . the results show context-dependent effects of multilingual training . |
Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that the flatness of the local minimum correlates well with better generalization. |
| Approach: | They propose to use a method encouraging convergence to a flatter minimum to fine-tune PLMs. |
| Outcome: | The proposed method outperforms state-of-the-art methods on NLP tasks without extra computation cost. |
Teach Small Models to Reason by Curriculum Distillation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Reasoning Models (LRMs) show strong System-2-style reasoning, but at the cost of significant computational overhead. |
| Approach: | They propose a two-stage curriculum distillation framework which builds a robust internal problem-solving student model and then teaches the student model to externalize this knowledge as explicit reasoning. |
| Outcome: | The proposed model outperforms single-stage baselines on mathematical benchmarks and significantly outperformed LRMs on complex tasks. |
Advancing Fine-Grained Visual Understanding with Multi-Scale Alignment in Multi-Modal Models (2025.emnlp-main)
Copied to clipboard
Wei Wang, Zhaowei Li, Qi Xu, Linfeng Li, YiQing Cai, Botian Jiang, Hang Song, Xingcan Hu, Pengyu Wang, Li Xiao
| Challenge: | Recent advances in multi-modal large language models have demonstrated remarkable capabilities in multimodal understanding, reasoning, and interaction. |
| Approach: | They propose a method that effectively aligns and integrates multi-scale knowledge of objects . they use a pipeline that provides over 300K essential training data to enhance alignment . |
| Outcome: | The proposed method effectively aligns and integrates multi-scale knowledge of objects, including texts, coordinates, and images. |
mALBERT: Is a Compact Multilingual BERT Model Still Worth It? (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies on the ethical and ecological impact of pre-trained language models raise questions about the temporal, financial, and environmental aspects of such models. |
| Approach: | They propose to focus on smaller models, such as compact models like ALBERT, which are more ecologically virtuous than these PLMs. |
| Outcome: | The proposed model is compared with classical multilingual models and is ethically virtuous. |
Targeted Distillation for Sentiment Analysis (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies demonstrate that large language models exhibit remarkable capabilities and achieve state-of-the-art performance in diverse sentiment analysis tasks. |
| Approach: | They propose a distillation framework that decouples knowledge from alignment and introduces a sentiment analysis benchmark that covers a diverse set of tasks. |
| Outcome: | The proposed framework improves models' generalization to unseen tasks and their generalization is strong against existing small-scale models. |
SLlama: Parameter-Efficient Language Model Architecture for Enhanced Linguistic Competence Under Strict Data Constraints (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large-scale language models (LLMs) have shown remarkable performance across a wide array of tasks. |
| Approach: | They propose an architecture that preserves parameter efficiency of tied models without sacrificing representational benefits of untied embeddings. |
| Outcome: | The proposed architecture achieves a 31.72% improvement in linguistic knowledge acquisition over the baseline model. |
RAG-Zeval: Enhancing RAG Responses Evaluator through End-to-End Reasoning and Ranking-Based Reinforcement Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation frameworks rely on direct prompting of resource-intensive models with complex multi-stage prompts, introducing significant computational cost and underutilizing models’ reasoning capabilities. |
| Approach: | They propose a framework that trains evaluators with reinforcement learning to generate comprehensive and sound assessments with detailed explanation in one-pass. |
| Outcome: | The proposed framework outperforms baseline evaluation frameworks that rely on LLMs with 10-100 more parameters and achieves the strongest correlation with human judgments. |
Dissecting Clinical Reasoning in Natural Language Inference for Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent studies on large language models (LLMs) have demonstrated the impact of prompting strategies and fine-tuning techniques on their reasoning capabilities. |
| Approach: | They examine four classes of prompting strategies to elicit reasoning in large language models . they then construct demonstrations using a frontier model to distil multi-step reasoning capabilities into smaller models based on Low-Rank Adaptation (LoRA). |
| Outcome: | The proposed model improves in 75% of the models on MedNLI and TREC Clinical Trials. |
Beyond Output Matching: Bidirectional Alignment for Enhanced In-Context Learning (2025.acl-long)
Copied to clipboard
Chengwei Qin, Wenhan Xia, Fangkai Jiao, Chen Chen, Yuchen Hu, Bosheng Ding, Ruirui Chen, Shafiq Joty
| Challenge: | Existing methods to train student models on the generated outputs of teacher models are not efficient for ICL. |
| Approach: | They propose to align the output of smaller (student) models with that of larger (teacher) models by incorporating a ranking loss and aligning the token-level output distribution. |
| Outcome: | The proposed model outperforms baseline models on a variety of tasks involving language understanding, reasoning, and coding. |
Can Compact Language Models Search Like Agents? Distillation-Guided Policy Optimization for Preserving Agentic RAG Capabilities (2026.acl-long)
Copied to clipboard
| Challenge: | Existing agentic RAG systems rely on large language models with billions of parameters. |
| Approach: | They propose a method to elicit agentic RAG behaviors from compact models . they propose ARC, which uses cold-start initialization and teacher guidance . |
| Outcome: | The proposed method outperforms the larger teacher model in some cases. |
Lightweight and Faithful Visual Condition Checking in Behavior Trees via Expert-Regularized Reinforcement Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing behavior trees are not suitable for high-dimensional perceptual inputs such as images or language. |
| Approach: | They propose a framework that leverages expert-regularized reinforcement learning to preserve semantic faithfulness while employing a factorized policy that aggregates sequential condition-node decisions into a single decision unit. |
| Outcome: | The proposed framework outperforms imitation learning and reinforcement learning but risks misalignment of condition nodes with intended semantics and poor credit assignment. |